Associated manuscript: Assessing the calibration of transition probabilities in a multistate model out of the initial state

Available at: https://github.com/manchester-predictive-healthcare-group/CHI-MRC-multi-outcome/tree/main/Project%206%20Calibration%20plots%20for%20multistate%20risk%20prediction%20models/manuscript%20files

This document contains all the supplementary Figures associated with the aforementioned manuscript. The supplementary material is split into 7 sections:

Section 1 - Large sample analysis: moderate calibration (RC: random censoring)

The first section of this document contains plots assessing the moderate calibration in the large development sample analysis for the pseudo-value and MLR-IPCW methods in the non-informative censoring (RC) scenario. To showcase each methods ability to appropriately assess non-linear patterns of miscalibration, there is a seperate plot for each method, containing the calibration plots for the perfectly calibrated, over predicting and under predicting transition probabilities. These plots are of the same type as Figure 2 from the main manuscript.

Figure S1: Assessment of moderate calibration for the BLR-IPCW approach in scenario RC, large sample analysis

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Figure S2: Assessment of moderate calibration for the pseudo-value approach in scenario RC, large sample analysis

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Figure S3: Assessment of moderate calibration for the MLR-IPCW approach in scenario RC, large sample analysis

Section 2 - Large sample analysis: moderate calibration (WAC and SAC: weakly and strongly associated censoring mechanism. Censoring mechanism is independent after adjustment on variables Z)

The second section of this document contains plots assessing the moderate calibration in the large development sample analysis for the BLR-IPCW, pseudo-value and MLR-IPCW methods in the weakly and strongly associated censoring scenarios (WAC and SAC). There is a seperate plot for each type of predicted transition probability, where all three methods (BLR-IPCW, pseudo-value and MLR-IPCW) are compared. These plots are of the same type as Figures 3 and 4 from the main manuscript.

Figure S4: Assessment of moderate calibration for each method

Scenario = WAC, Perfectly calibrated transition probabilities---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Figure S5: Assessment of moderate calibration for each method

Scenario = WAC, Miscalibrated 1---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Figure S6: Assessment of moderate calibration for each method

Scenario = WAC, Miscalibrated 2---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Figure S7: Assessment of moderate calibration for each method

Scenario = SAC, Perfectly calibrated transition probabilities---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Figure S8: Assessment of moderate calibration for each method

Scenario = SAC, Miscalibrated 1---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Figure S9: Assessment of moderate calibration for each method

Scenario = SAC, Miscalibrated 2

Section 3 - Large sample analysis: Mean calibration

The mean calibration according to AJ, BLR-IPCW and MLR-IPCW is presented for the perfectly calibrated, and miscalibrated predicted transition probabilities. This is Figure 4 from the manuscript.

Figure S10: Large sample analysis, mean calibration

Section 4 - Small sample analysis: moderate calibration

This section contains the moderate calibration plots for methods BLR-IPCW and pseudo-value in the small sample analysis. The calibration curves from 200 simulation iterations are superimposed on top of eachother. Given MLR-IPCW was a scatter plot, we could not devise a suitable way to present the scatter plots across the 200 simulation iterations. This is akin to the problem that it is unclear how to present sampling uncertainty (i.e. a confidence interval) for the calibration scatter plots derived from MLR-IPCW, whereas confidence intervals can be estimated and presented for the calibraion curves from the BLR-IPCW and pseudo-value approaches (e.g. see results from large sample analysis, moderate calibration).

Figure S11: Assessment of moderate calibration for BLR-IPCW and pseudo-value approach in the small sample analysis

Scenario = RC, Perfectly calibrated transition probabilities, N = 1500

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Figure S12: Assessment of moderate calibration for BLR-IPCW and pseudo-value approach in the small sample analysis

Scenario = RC, Miscalibrated 1, N = 1500

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Figure S13: Assessment of moderate calibration for BLR-IPCW and pseudo-value approach in the small sample analysis

Scenario = RC, Miscalibrated 2, N = 1500

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Figure S14: Assessment of moderate calibration for BLR-IPCW and pseudo-value approach in the small sample analysis

Scenario = RC, Perfectly calibrated transition probabilities, N = 3000

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Figure S15: Assessment of moderate calibration for BLR-IPCW and pseudo-value approach in the small sample analysis

Scenario = RC, Miscalibrated 1, N = 3000

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Figure S16: Assessment of moderate calibration for BLR-IPCW and pseudo-value approach in the small sample analysis

Scenario = RC, Miscalibrated 2, N = 3000

Section 5 - Small sample analysis: mean calibration

This section contains the mean calibration plots (median and 2.5 - 97.5 percentile range across 1,000 simulation iterations) for the small sample analysis.

Figure S17: Small sample analysis. Median and 2.5 - 97.5 percentile range in bias of mean calibration. N = 3000, groups = 10.

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Figure S18: Small sample analysis. Median and 2.5 - 97.5 percentile range in bias of mean calibration. N = 1500, groups = 10.

Section 6 - Sensitivity analyses

The third section of this document contains sensitivity analyses. We assess performance of BLR-IPCW and MLR-IPCW without the inverse probability of censoring weights applied, and when the weights are perfectly specified, i.e. taken from the data generating mechanism as opposed to estimated from the data. We assess performance of the pseudo-value approach without grouping individuals by predicted risk before estimating the pseudo-values. This section is divided into 5 parts.

Section 6.1 - Sensitivity analyses for BLR-IPCW in large sample analysis, moderate calibration

We assess performance of BLR-IPCW without the inverse probability of censoring weights applied, and when the weights are perfectly specified.

Figure S19: Misspecification of weights, BLR

Scenario = RC, Perfectly calibrated transition probabilities---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Figure S20: Misspecification of weights, BLR

Scenario = RC, Miscalibrated 1---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Figure S21: Misspecification of weights, BLR

Scenario = RC, Miscalibrated 2---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Figure S22: Misspecification of weights, BLR

Scenario = WAC, Perfectly calibrated transition probabilities---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Figure S23: Misspecification of weights, BLR

Scenario = WAC, Miscalibrated 1---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Figure S24: Misspecification of weights, BLR

Scenario = WAC, Miscalibrated 2---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Figure S25: Misspecification of weights, BLR

Scenario = SAC, Perfectly calibrated transition probabilities---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Figure S26: Misspecification of weights, BLR

Scenario = SAC, Miscalibrated 1---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Figure S27: Misspecification of weights, BLR

Scenario = SAC, Miscalibrated 2

6.2 Sensitivity analyses for MLR-IPCW in large sample analysis, moderate calibration

We assess performance of MLR-IPCW without the inverse probability of censoring weights applied, and when the weights are perfectly specified.

Figure S28: Misspecification of weights, MLR

Scenario = RC, Perfectly calibrated transition probabilities---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Figure S29: Misspecification of weights, MLR

Scenario = RC, Miscalibrated 1---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Figure S30: Misspecification of weights, MLR

Scenario = RC, Miscalibrated 2---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Figure S31: Misspecification of weights, MLR

Scenario = WAC, Perfectly calibrated transition probabilities---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Figure S32: Misspecification of weights, MLR

Scenario = WAC, Miscalibrated 1---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Figure S33: Misspecification of weights, MLR

Scenario = WAC, Miscalibrated 2---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Figure S34: Misspecification of weights, MLR

Scenario = SAC, Perfectly calibrated transition probabilities---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Figure S35: Misspecification of weights, MLR

Scenario = SAC, Miscalibrated 1---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Figure S36: Misspecification of weights, MLR

Scenario = SAC, Miscalibrated 2

6.3 Sensitivity analyses for pseudo-value method in large sample analysis, moderate calibration

We assess performance of the pseudo-value approach without grouping individuals by predicted risk before estimating the pseudo-values.

Figure S37: Not grouping individuals by predicted risk of each state, pseudo-value method

Scenario = RC, Perfectly calibrated transition probabilities---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Figure S38: Not grouping individuals by predicted risk of each state, pseudo-value method

Scenario = RC, Miscalibrated 1---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Figure S39: Not grouping individuals by predicted risk of each state, pseudo-value method

Scenario = RC, Miscalibrated 2---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Figure S40: Not grouping individuals by predicted risk of each state, pseudo-value method

Scenario = WAC, Perfectly calibrated transition probabilities---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Figure S41: Not grouping individuals by predicted risk of each state, pseudo-value method

Scenario = WAC, Miscalibrated 1---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Figure S42: Not grouping individuals by predicted risk of each state, pseudo-value method

Scenario = WAC, Miscalibrated 2---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Figure S43: Not grouping individuals by predicted risk of each state, pseudo-value method

Scenario = SAC, Perfectly calibrated transition probabilities---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Figure S44: Not grouping individuals by predicted risk of each state, pseudo-value method

Scenario = SAC, Miscalibrated 1---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Figure S45: Not grouping individuals by predicted risk of each state, pseudo-value method

Scenario = SAC, Miscalibrated 2

Section 6.4 - Sensitivity analyses for large sample analysis, mean calibration

We now present sensitivity analyses for the large sample analysis assessment of mean calibration. The plot is the same as that in section 3 in this document, except AJ is implemented without grouping individuals by predicted risk, and BLR-IPCW and MLR-IPCW are implemented without inverse probability of censoring weights.

Figure S46: Large sample analysis, mean calibration, sensitivity analysis. AJ implemented without grouping individuals by predicted transition probabilities of state of interest. BLR-IPCW and MLR-IPCW implemented without inverse probability of censoring weights.

Section 6.5 - Sensitivity analyses for small sample analysis, mean calibration

We then present sensitivity analyses for the small sample analysis assessment of mean calibration. The plots are the same as those in section 5 in this document, except AJ is implemented without grouping individuals by predicted risk, and BLR-IPCW and MLR-IPCW are implemented without inverse probability of censoring weights.

Figure S47: Small sample analysis, sensitivity analysis. Median and 2.5 - 97.5 percentile range in bias of mean calibration. N = 3000. AJ implemented without grouping individuals by predicted transition probabilities of state of interest. BLR-IPCW and MLR-IPCW implemented without inverse probability of censoring weights.

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Figure S48: Small sample analysis, sensitivity analysis. Median and 2.5 - 97.5 percentile range in bias of mean calibration. N = 1500. AJ implemented without grouping individuals by predicted transition probabilities of state of interest. BLR-IPCW and MLR-IPCW implemented without inverse probability of censoring weights.

Section 7 - Clinical example

This section contains the moderate calibration plot for the clinical example, when using a development dataset of size N = 100,000. This model, and the model from the main manuscript (N = 5,000) were both validated in the same validation dataset of size N = 100,000. The closer grouping of points in the MLR-IPCW calibration scatter plot is evident for the model with development sample size N = 100,000.

Figure S49: Moderate calibration according to each method (development sample size N = 100,000)